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Abstract 

Validating a concept of operation for a complex, safety-critical system (like the National Airspace Sys- 
tem) is challenging because of the high dimensionality of the controllable parameters and the infinite number 
of states of the system. In this paper, we use statistical modeling techniques to explore the behavior of a 
conflict detection and resolution algorithm designed for the terminal airspace. These techniques predict the 
robustness of the system simulation to both nominal and off-nominal behaviors within the overall airspace. 
They also can be used to evaluate the output of the simulation against recorded airspace data. Additionally, 
the techniques carry with them a mathematical value of the worth of each prediction-a statistical uncertainty 
for any robustness estimate. Uncertainty Quantification (UQ) is the process of quantitative characterization 
and ultimately a reduction of uncertainties in complex systems. UQ is important for understanding the 
influence of uncertainties on the behavior of a system and therefore is valuable for design, analysis, and veri- 
fication and validation. In this paper, we apply advanced statistical modeling methodologies and techniques 
on an advanced air traffic management system, namely the Terminal Tactical Separation Assured Flight 
Environment (T-TSAFE). We show initial results for a parameter analysis and safety boundary (envelope) 
detection in the high-dimensional parameter space. For our boundary analysis, we developed a new sequen- 
tial approach based upon the design of computer experiments, allowing us to incorporate knowledge from 
domain experts into our modeling and to determine the most likely boundary shapes and its parameters. We 
carried out the analysis on system parameters and describe an initial approach that will allow us to include 
time-series inputs, such as the radar track data, into the analysis. 

I. Introduction 

NASA is developing new tools and procedures that are intended to improve the safety of the nation’s 
air transportation system and to replace the legacy conflict prediction tools in place today. One promising 
NASA concept of operation, the Tactical Separation Assured Flight Environment (TSAFE), works in the en 
route airspace and uses both flight intent information and dead reckoning to calculate trajectories. 1,2 TSAFE 
was specifically designed for conflict detection en route, and is part of a larger airspace management system 
known as the Automated Airspace Concept . 3 Detecting conflicts within the terminal airspace can be more 
complex than detection within en route airspace, for reasons we will discuss below. NASA has developed 
a tactical conflict detection and resolution tool specifically designed for the complexities of the terminal 
airspace that is based on TSAFE, called the Terminal Tactical Separation Assured Flight Environment, or 
Terminal TSAFE (T-TSAFE). 4 

An important safety goal we focus on in this work is to maintain safe separation between all aircraft. 
Maintaining separation in the terminal airspace (near an airport) is much more challenging than maintaining 
separation in en route airspace. In the terminal airspace there are higher rates of operational errors, denser 
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air traffic, frequent large turns, incompletely specified flight plans, complex sets of separation standards, and 
the frequent necessity to operate aircraft purposely near the required separation standards. Terminal airspace 
surrounds airports within a radius of about 40 miles. Within terminal airspace, the legal separation standards 
are complex, and depend on aircraft weight class, type of approach, visual vs. instrument flight rules, and 
whether the aircraft is transitioning to or from en route airspace. Terminal area air traffic controllers guide 
aircraft as they approach or depart the airport, and must maintain these separation standards. However, 
the complexity of terminal airspace has proven difficult for the tactical conflict detection systems that are 
intended to aid terminal area air traffic controllers. 

The Terminal Tactical Separation Assured Flight Environment (T-TSAFE) that is being developed at 
NASA Ames is designed to overcome these difficulties. T-TSAFE creates a single predicted trajectory for 
each aircraft based on an algorithm that uses available flight intent information. The flight intent information 
includes flight plans, area navigation departure routes, site-specific nominal arrival routes, speed restrictions, 
and altitude clearances. Additionally, the separation algorithms in T-TSAFE are governed by a large number 
of configuration, aircraft performance (e.g., aircraft weight), and operational parameters. T-TSAFE also 
includes a refined set of current, dynamic separation standards for terminal airspace to define losses of 
separation. By combining all of these variables, T-TSAFE is able to predict the future positions of aircraft 
and check them for possible conflicts with significantly fewer false alerts than the current legacy system. 4 

Obviously, it is important to ensure T-TSAFE is working correctly and produces as few prediction 
mistakes as possible, even under off-nominal conditions or when the normal behavior of the air traffic system 
changes. Our goal is to work towards the validation of T-TSAFE within this safety-critical system. In 
particular, we want to a.) make sure that the system is robust to extreme input configurations and also to 
b.) quantify the effect of uncertainties in the system on the predictions. In order to ensure safe operation 
through validation, we must analyze how uncertainties and noise in those parameters influence the behavior 
of T-TSAFE. However, current validation techniques for such systems are usually limited to exploring a 
small set of manually selected individual senarios. 

Uncertainty Quantification (UQ) is an active area of science that seeks to understand the sources of 
prediction error in a complex system, and further to quantify and to hopefully reduce these sources of 
uncertainty in order to achieve better predictions about the complex system’s behavior. 5,6 

UQ is important for understanding the influence of uncertainties in the behavior of the system and 
therefore is very valuable for design, analysis and, in particular for the focus of our work, validation of 
the system. In order to carry out a UQ analysis for T-TSAFE, it is important to understand interactions 
between continuous variables (variables which can have infinitely small variation within their range) , discrete 
variables (variables which can only take on 1 of n finite values), and time series variables (variables which can 
change their values with time). Systems which depend on a combination of continuous and discrete variables 
for their behavior are known as hybrid systems and require a substantial extension to the current state of 
the art for a complete analysis.'’ 8 In this paper, we carry out an initial UQ analysis for T-TSAFE; our 
initial results give insight about those configuration parameters which are actually influential on T-TSAFE’s 
capability to detect losses of separation between two aircraft. 

A. Separation Assurance 

The National Airspace (NAS) is at its performance limit and the number of flights may increase by a factor 
of 2 or more by the year 2020. 9,10 Researchers on Next Generation Air Traffic Control develop approaches 
and systems to safely handle the increased number of aircraft in the NAS by using increasingly automated 
systems. The important safety goal is to maintain safe separation between all aircraft: usually 3 nmi 
horizontal and 1000 feet vertical, referred to as “the hockey puck” (see Figure 1). The terminal airspace 
near an airport is much more challenging in terms of density and complexity than en route airspace, and 
operating errors are higher. The complexity of terminal airspace has proven difficult for tactical conflict 
detection systems. Contributing factors to this complexity include the dense air traffic, frequent large turns, 
incompletely specified flight plans, a complex set of separation standards, and the frequent necessity to 
operate aircraft purposely near the required separation standards. 

Accurate and early prediction of any loss of separation (LOS) is necessary to support the controller or for 
automated systems. There are three independent layers of separation assurance. The strategic layer focuses 
on mid-term conflicts (losses of separation) predicted to be between 2 and 20 minutes into the future. The 
tactical layer addresses short-term or imminent conflicts predicted to occur within 2 minutes. A third layer 
of safety is provided by an independent airborne collision avoidance system such as TCAS (Traffic Alert 
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Figure 1. The aircraft separation “hockey-puck” 11 


and Collision Avoidance System). TCAS deals with potential collisions less than 45 seconds away. The 
tactical layer for en route flights, known as the Tactical Separation Assured Flight Environment (TSAFE), 
is proposed as a backup system that duplicates a limited set of safety-critical functions of the strategic layer. 
TSAFE simplifies the problem of automated separation assurance and provides a safety net for the strategic 
layer. Algorithms for the prediction of LOS on all layers are safety-critical and thus require careful analysis 
and validation. 

B. Details of the T-TSAFE System 

The Terminal Tactical Separation Assured Flight Environment (T-TSAFE) predicts LOS near airports for 
take-offs and landings. 12 T-TSAFE encodes all complex terminal separation rules including wake separation. 
T-TSAFE uses a novel intent-based algorithm for predicting the trajectories of the aircraft for the next 2 
minutes and is implemented in the Java computer language. 

The architecture of T-TSAFE is shown in Figure 2. T-TSAFE uses different kinds of inputs: static 
configuration data (TSAFE properties), static aircraft data (e.g., aircraft weight, climb speed, and other 
performance data as given in the Base of Aircraft Data (BADA) 13, 14 ), and the actual aircraft track data 
including weather information. Track data and weather are time series data. Track data, stored in CTAS’s 
cm-sim data format contain the aircraft’s ID and flight number, position, ground speed, altitude, and addi- 
tional data in time intervals of 12 seconds (one ATC radar sweep). The weather data contain wind speed, 
direction, temperature, and air pressure for a certain location in a 5 km grid and altitude for each hour of 
the day. 

After reading the static data, T-TSAFE processes the track data and reports potential conflicts with 
time-to-loss (ttlos) and performs predictions in regular intervals. A utility function read_conf licts reads 
those reported conflicts and produces a data structure for further processing. 

Figure 3 shows a typical scenario with two aircraft. The track data originate from an actual LOS scenario 
in which we have obscured the airline information. The aircraft track for flight ABC653 is shown in blue, 
for XYZ1052 in green. Both aircraft are in final descent. The figure shows the world coordinates in nautical 
miles (nmi) in the x and y directions and the altitude in thousands of feet. The red lines connect the 
trajectories at points in which there were actual losses of separation; when both aircraft came too close 
together and violated the terminal separation rules. This figure shows that several losses of separation are 
detected (numlos); the first loss of separation occurs at time ttlos. 

C. Statistical Modeling for T-TSAFE 

In our study, we investigated how T-TSAFE copes with uncertainties and trajectory prediction (TP) errors. 
In general, uncertainties can occur due to: 

• intent errors and delays (e.g. pilot doesn’t follow ATC commands), 

• errors in the initial conditions (e.g. radar measurements), 
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Figure 2. Architecture of TTSAFE 



Figure 3. 


Two aircraft scenario, points at which the aircraft lose separation are connected with red lines. 


• environmental uncertainties (e.g. weather), 

• and modeling errors (e.g. unknown actual weight of aircraft). 

We analyzed how the specific separation assurance algorithm in T-TSAFE behaves in the presence of these 
uncertainties. In general, we consider that all three kinds of inputs to T-TSAFE as shown in Figure 2 can 
contain sources of uncertainties. However, in this paper we report only on the uncertainty with respect to 
the configuration inputs and to the track data. We will present a more thorough analysis in future work. We 
perform both time series analysis and boundary analysis within our study, using techniques only recently 
developed within the statistics community. 15 In the following section, we first give a short introduction into 
Uncertainty Quantification (UQ). We then overview sensitivity analysis and briefly discuss our boundary 
estimation technique. Finally, we describe and report on our experimental in the subsequent sections. 
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II. Uncertainty Quantification 


A. Concepts 

Uncertainty Quantification(UQ) is the science of quantitative characterization and reduction of uncertainties 
in systems. UQ is increasingly becoming an important part of design and analysis of complex computer 
and engineering systems. 16-18 It provides the ability to assess the quality of computational results and 
apply confidence bounds to output metrics, particularly problems in which experimental data is difficult or 
impossible to obtain. 

Uncertainties from numerous sources can affect the behavior of a complex system, such as: 

• Parameter uncertainty, which comes from the model parameters that are inputs to the computer 
model, but whose exact values are unknown to experimentalists and cannot be controlled in physical 
experiments; 

• Structural uncertainty, aka model inadequacy, model bias, or model discrepancy, which comes from 
the lack of knowledge of the underlying true physics. It depends on how accurately a mathematical 
model describes the true system for a real-life situation, considering the fact that models are almost 
always only approximations to reality; 

• Algorithmic uncertainty, aka numerical uncertainty, which comes from numerical errors and numerical 
approximations per implementation of the computer model. Most models are too complicated to solve 
exactly; 

• Experimental uncertainty (observation error) which comes from the variability of experimental mea- 
surements. UQ is valuable for design analysis and for validation. 

In general, there are two types of problems in UQ analysis: forward uncertainty propagation is used for 
the analysis of system outputs based on uncertain inputs; the inverse problem is used to assess model and 
parameter uncertainty based on the outputs. 



Figure 4. Uncertainty Quantification 


B. Sensitivity Analysis 

For the parameter analysis, we are interested in studying how the uncertainty in the output of the system 
can be apportioned to different sources of uncertainty in its inputs. To do so, we performed sensitivity 
analysis on the parameters, and usually uncertainty and parameter sensitivity analysis are run in tandem. 
Sensitivity analysis can be useful for a range of purposes, some examples include: 

• Testing the robustness of the system in the presence of uncertainty, 

• Identifying model inputs that cause significant uncertainty in the output thus should therefore be the 
focus of further analysis, 

• Model simplification i.e. fixing model inputs that have no effect on the output, 

• Efficient searching for errors in the model. 

In models involving many input variables, parameter analysis is an essential ingredient of model building 
and quality assurance. 
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III. Experiments and Results 


A. Experimental Setup: T-TSAFE Harness 

For our statistical analysis, we consider T-TSAFE strictly as a black box, i.e. , we do not have access to any 
of the internal details and variables of the T-TSAFE system. For each experimental trial, we provide T- 
TSAFE with perturbed variables to its configuration parameters and to the track data, as shown in Figure 2. 
Each of the inputs is parameterized, i.e., it can deviate in a certain way from its original, or nominal value. 
Table 1 gives the values perturbed for the static configuration and we discuss our track perturbation below in 
subsection C. As the driving scenario, we use — for this study — the two-aircraft scenario shown in Figure 3: 
flights ABC653 and XYZ1052 losing safe separation during their descent. 

Figure 5 shows our experimental setup. In a first step, the variables and parameters that need to be 
perturbed (see above) must be specified by the user. Typically, a range can be given, e.g., the time interval 
for invoking the detection algorithm is between 5 seconds and 200 seconds, or variations around the nominal 
value, e.g., the aircraft weight w can be perturbed by ±25%, i.e., w = wq ± 25%. 

Based upon this specification, our harness generates a number of test cases. Each test case corresponds 
to a specific set of inputs. With those inputs the TTSAFE system is started, and the outputs of interest, 
mainly the variables numlos, ttlos and the position and altitude of the first detection of LOS ( x , y , alt) are 
extracted using the utility function read_conf licts. After all relevant test cases have been processed, we 
perform the statistical data analysis and visualize the results. 



Variables to modify 


Statistical Data Analysis 


Visualization 


Figure 5. Architecture of the TTSAFE Harness 


B. Sensitivity Analysis on Configuration Inputs 

For the analysis on configuration inputs, we used 14 scalar configuration parameters as shown in Table 1. 
Of those, 10 are continuous and 4 variables are discrete. With this parameter definition, we used a 3-factor 
combinatorial exploration 19-21 to generate 2467 distinct test cases. For continuous variables, a uniform 
distribution within a range of ±25% was approximated using 5 equidistant bins. The discrete configuration 
parameters are used to select between varying algorithms for detecting conformance and predicting aircraft 
trajectories. We executed T-TSAFE on each of the 2467 test cases and we recorded numlos and ttlos. 

For the given experiments and each of the parameter variables, we calculated the standardized regression 
coefficient (SRC), a commonly used sensitivity value. Figure 6 shows the corresponding plot. Small circles 
show the SRC values, the short lines indicate their variance (which is very low in this case). This plot 
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Parameter Name 

Nominal Value 

Type 

MinHorizSepNmi 

3 

C 

aboveFL290AltLevel 

1000 

C 

aboveFL290AltChanging 

1000 

C 

belowFL290AltLevel 

1000 

C 

belowFL290AltChanging 

1000 

C 

CheckPeriodMinutes 

1.0 

C* 

CrossTrackTolerance 

0.5 

C 

AlongTrackTolerance 

0.25 

C 

AltitudeTolerance 

200 

C 

HeadingTolerance 

5 

C 

ConformanceStrategy 

No default 

D(2) 

ConflictDetection 

No default 

D(3) 

NomTrajSynth 

No default 

D(3) 

DeadReckonTS 

No default 

D(2) 


Table 1. TTSAFE configuration parameters perturbed in our experiments. Continuous parameters (C) were 
perturbed with a range of d=25% of the nominal value. D(:r) denotes a discrete variable with x distinct values. 


indicates that only three variables ( MinHorizSepNmi, CheckPeriodMinutes, and ConflictDetection) have 
SRC values which are substantially larger than 0, meaning that the behavior of T-TSAFE is sensitive to 
those variables. 


SRC 



MinHorizSepNmi CheckPeriodMinutes HeadingTolerance DeadReckonTS 


Figure 6. Sensitivity Analysis Results. Values near zero indicate that the output is insensitive to the input. 

Some of the results make sense. For example, MinHorizSepNmi (the minimum allowable horizontal 
separation in nautical miles) governs if there is a LOS or not, so T-TSAFE should be sensitive to that input. 
Other results, such as the fact that T-TSAFE is relatively insensitive to the algorithms used for trajectory 
prediction, are less intuitive. It may be that a more sophisticated exploration in which the trajectories of 
the aircraft are significantly varied would produce a different result. Figure 7 shows the scatter plots for 
mimlos, supporting the results of the sensitivity analysis. 
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AllitudeT olerance 


C onf otmanc e Strat e gy 


ContlictD election 


DeadReckonTS 


Figure 7. Scatterplots for numlos 


C. Uncertainty in Track Data 

For this experiment, uncertainties in the measured track data of both aircraft were analyzed. These track 
data are generally obtained by measurements of ATC radar and can be sources of substantial errors and 
uncertainties. 

These data are time series data and a number of different possibilities exist to perturb them for uncertainty 
analysis. Here, we pick a simple, yet relevant mechanism: each of the track data are offset by a given, fixed 
bias. Such a perturbation can, for example, correspond to a systematic bias in the radar-based altitude 
measurements . 

More specifically, for this experiment, we modified the altitude by a uniformly distributed time-constant 
bias S € {— 1000/f, 1000/t} for each aircraft. Figure 8 shows the behavior of TTSAFE over the bias 5ac\ 
and 5aC2- The nominal case is thus in the middle of this figure. Individual trials are shown in magenta for 
numlos < 4 (i.e. , no losses detected), red for 4 < numlos < 7, and blue otherwise. Obviously the triangular 
shapes are caused by the geometry of the scenario: if the bias causes the aircraft be too far away from each 
other vertically, then fewer losses are detected. The asymmetry between AC1 and AC2 might be of interest 
for further investigation as well as a “red” region surrounded by blue in the vicinity of 5aci = —800... — 300 
and Sac 2 = —200... 200. 

A similar visualization for ttlos is shown in Figure 9. The same trials as discussed above show quite 
a different picture: ttlos keeps its nominal value except for a tiny area in the parameter space. Trials are 
marked in red for ttlos < 0.5. 

This small (appr. 125x125ft) area could be of interest for design and validation as it shows a high 
sensitivity and is located in the “middle” of an otherwise smooth region. A zoomed-in plot is shown in 
Figure 9 (right), revealing a triangular shape. 

IV. Conclusions 

In this paper, we show an initial uncertainty quantification analysis for an air traffic management concept 
of operation (T-TSAFE). This analysis shows that there is a wide variation in the effect the input configu- 
ration parameters have on the ability of T-TSAFE to detect losses of separation, as seen in Figure 6. It also 
revealed an unexpected region of behavior that can be affected by an altitude measurement bias, as shown 
in Figure 8. We believe that these sorts of analyses are invaluable to the design and analysis of complex 
systems, particularly those that are safety-critical. 

In our future work, we plan to: 

• extend current methodologies and perform further analysis for hybrid inputs consisting of continuous 
and discrete parameters, 
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magenta: numlos < 4, red: numlos < 7 



Figure 8. Different values of numlos over different altitude biases for AC1 and AC2 


• use this extended analysis technique to study additional TTSAFE outputs — e.g. estimated points of 
LOS, aircraft conformance, 

• perform uncertainty analyses for aircraft parameters such as weight and performance, e.g., study the 
influence of weight on climb and descent speed, 

• further study the influence of track uncertainty (time-series data), 

• use multi-aircraft scenarios for the analysis of missed-alarm/false-alarm rates. 
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